Drunk Men has put together the results of a one-year long experiment to monitor spider activity from the major three search engines. If you’re watching crawling performance on big content sites, this is must read. Note their graph (in log-based scale) of relative levels of performance over time:
It may come as a surprise to many in SEO that Yahoo! is the fastest and deepest crawling bot for this content. I’d argue that in the last 2-3 months, I’ve been seeing MSN overtake Yahoo!’s crawler in both depth and breadth, and I’ve also seen Google, once induced by strong external links, become the most rampant and hungry (if not speediest) bot of all.
My favorite part of their results is actually this graphical representation of how Yahoo! crawled:
Like a beautiful, black hat fractal…
